Skip to content

gh-153837: Fix IndentationError offset in tokenize to point to the indentation error, not end of line#154281

Closed
ghost wants to merge 1 commit into
mainfrom
unknown repository
Closed

gh-153837: Fix IndentationError offset in tokenize to point to the indentation error, not end of line#154281
ghost wants to merge 1 commit into
mainfrom
unknown repository

Conversation

@ghost

@ghost ghost commented Jul 20, 2026

Copy link
Copy Markdown

Since Python 3.12 (PEP 701 refactoring), the module reports incorrect column offsets on and related indentation errors. The function in unconditionally used (the end of the current line) as the column offset, producing an offset equal to the line length rather than pointing to where the indentation error actually occurs.

For indentation errors (E_DEDENT, E_TABSPACE, E_TOODEEP), this fix computes the offset by scanning from the start of the line to the first non-whitespace character, which is where the indentation error is detected. This restores the correct behavior that existed in Python 3.11 and earlier, where the offset pointed to the problematic indentation rather than the end of the line.

@ghost
ghost requested review from lysnikolaou and pablogsal as code owners July 20, 2026 18:05
@bedevere-app

bedevere-app Bot commented Jul 20, 2026

Copy link
Copy Markdown

Most changes to Python require a NEWS entry. Add one using the blurb_it web app or the blurb command-line tool.

If this change has little impact on Python users, wait for a maintainer to apply the skip news label instead.

@ghost
ghost force-pushed the fix-gh-153837-tokenize-indent-offset branch 2 times, most recently from 1f9d26c to 55ff05e Compare July 20, 2026 18:36
@ghost
ghost force-pushed the fix-gh-153837-tokenize-indent-offset branch from 55ff05e to fc1b106 Compare July 20, 2026 20:04
Repository owner closed this by deleting the head repository Jul 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant